In the case of capturing an image by a digital camera, noise may sometimes be added to the image due to the characteristics of a CCD (Charge-Coupled Device) or a readout circuit for CMOS or the characteristics of transmission paths. Also, blur of an image due to an out-of-focus condition in capturing of the image or blur of an image due to camera shake occurs. In this way, the captured image has blur which is attributed to user's handling of the camera in photographing in addition to the noise which is attributed to the specific characteristics of the captured image, so that the image is degraded. Among such “blur” types, blur of an image which is attributed to a motion of a camera during photographing (exposure) is herein referred to as “motion blur”, so that it is distinguishable from blur that is attributed to an out-of-focus condition (out-of-focus blur).
In recent years, especially, demand for high sensitive photography is increasing, and therefore, it is necessary to restore an image degraded by blur (hereinafter, “degraded image”) to an image which is as close to an original image (hereinafter, “ideal image”) as possible. To realize a bright image which is free from noise or blur, such as an image demanded in high sensitive photography, the solutions are generally divided into two ideas, increasing the sensitivity and extending the exposure time.
However, increasing the sensitivity disadvantageously leads to amplification of noise. As a result, a signal is buried in the noise so that, in many cases, large part of a resultant image is formed by the noise. On the other hand, extending the exposure time enables accumulation of a larger amount of light which occurs at that site, resulting in an image which includes smaller noise. In this case, a signal would not be buried in the noise, but there is a problem of generation of motion blur in an image due to camera shake.
According to the prior art, there are two countermeasures against the problem resulting from the extended exposure time. One is optical camera shake compensation, such as lens shift, sensor shift, or the like. The other one is obtaining the direction/magnitude of motion blur from a resultant image and performing signal processing based on the obtained direction/magnitude of the blur to restore the image (a restoration method based on signal processing). The restoration method based on signal processing is, for example, disclosed in Patent Document 1, Patent Document 2, and Non-patent Documents 1 to 5.
A phenomenon that an image is degraded due to camera shake, from an ideal image to a degraded image, can be modeled as described below. A function which represents the brightness of each pixel of the degraded image is thought to be obtained by convolution of a function that represents the brightness of each pixel in the ideal image and a point spread function (PSF) that represents blur of an image which is caused by camera shake during photographing of the image. Restoration of the ideal image based on the obtained degraded image is realized by deconvolution of the degraded image and the PSF. A convolution operation is a multiplication in the frequency space. Therefore, in the frequency space, the degraded image is divided by the PSF, whereby the restored image can be obtained.
Thus, when the PSF is known, the restored image can be obtained relatively readily by means of the above-described deconvolution so long as the effect of noise is neglected. On the other hand, when the PSF is unknown, it is necessary to estimate the PSF from the degraded image in order to obtain a restored image. Estimation of the PSF may be realized by, for example, a method based on the sparse coding concept which is disclosed in Non-patent Document 1.
According to this method, in the first place, a first restoration result is obtained from a manually-given initial PSF and a degraded image. Then, the first restoration result and the degraded image are used to estimate a PSF which is close to a true PSF. The initial PSF is amended with the estimated PSF. The amended PSF is used to obtain the second restoration result based on the degraded image. Subsequently, the step of obtaining the Nth restored image from the (N−1)th PSF and the degraded image and the step of estimating the Nth PSF from the Nth restored image and the degraded image are repeated, whereby a PSF estimation process and a restoration process performed on the degraded image are concurrently advanced.